Semantic consistency of explanations in explainable artificial intelligence applications

ABSTRACT

A computer-implemented method in accordance with one embodiment includes, in response to a submission of an input dataset to an artificially intelligent application, receiving an explanation from each module of the application. The modules are configured within the application in a serial sequence in which each module, upon receiving the input dataset and any input generated by an immediately preceding module of the serial sequence, generates output that is forwarded as input to a next module, if any, in the sequence. A determination is made that at least two of the received explanations are semantically inconsistent.

BACKGROUND

The present invention relates in general to Explainable ArtificialIntelligence (“explainable AI” or “XAI”) technology and in particular toresolving semantic inconsistencies among explanations generated by XAIsystems.

Explainable AI refers to applications of artificial intelligencetechnology (AI), such as certain expert systems, that explain theresults they produce in a form that can be understood by humans. XAIaugments conventional “black box” AI and machine-learning methodologiesin which even a designer of an artificially intelligent applicationcannot determine why the application arrived at a particular decision.

SUMMARY

Embodiments of the present invention comprise systems, methods, andcomputer program products for an improved explainable artificiallyintelligent (XAI) application.

An XAI system in accordance with one embodiment includes a processor, amemory coupled to the processor, and a computer-readable hardwarestorage device coupled to the processor. The storage device containsprogram code configured to be run by the processor to implement amethod. The method includes, in response to a submission of an inputdataset to an artificially intelligent application, receiving, by theprocessor, explanations from modules of the application. The modules areconfigured within the application in a serial sequence in which eachmodule, upon receiving the input dataset and any input generated by animmediately preceding module of the serial sequence, generates outputthat is forwarded as input to a next module, if any, in the sequence.Each explanation of the received explanations explains an artificiallyintelligent reasoning by which a corresponding output is generated by acorresponding module of the application. A determination is made that atleast two of the received explanations are semantically inconsistent.

A computer-implemented method in accordance with one embodimentincludes, in response to a submission of an input dataset to anartificially intelligent application, receiving an explanation from eachmodule of the application. The modules are configured within theapplication in a serial sequence in which each module, upon receivingthe input dataset and any input generated by an immediately precedingmodule of the serial sequence, generates output that is forwarded asinput to a next module, if any, in the sequence. A determination is madethat at least two of the received explanations are semanticallyinconsistent.

A computer program product in accordance with one embodiment includes acomputer readable storage medium having program instructions embodiedtherewith, the program instructions being executable by a processor tocause the processor to perform a method. The method includes receiving,by the processor, an explanation from each module of an artificiallyintelligent application, the explanations being created by the modulesin response to a submission of an input dataset to the application. Themodules are configured within the application in a serial sequence inwhich each module, upon receiving the input dataset and any inputgenerated by an immediately preceding module of the serial sequence,generates output that is forwarded as input to a next module, if any, inthe sequence. A determination as made that at least two of the receivedexplanations are semantically inconsistent.

In another embodiment, the application contains an ordered sequence ofartificially intelligent software modules. When an input dataset issubmitted to the application, each module generates an output datasetand explanations that represent, as a set of Boolean expressions,reasoning by which each element of the output dataset was chosen by thecorresponding module. Each module's output is a set of elementsgenerated as a function of the input dataset and of output generated bythe module, if any, that is immediately preceding in the sequence. Ifany pair of explanations are confirmed to be semantically inconsistent,nonzero inconsistency scores are assigned to the inconsistent elementsof the explanations. Corresponding elements of a pair of explanationsare deemed to be inconsistent if the Boolean expressions embodied by theexplanations evaluate to FALSE conditions or if the explanations'corresponding modules produced disjoint outputs in response to theparticular contents of the input dataset. The system confirms whetherexplanations found to be semantically inconsistent by this method are infact inconsistent or merely appear to be anomalous because the inputdataset contained an unexpected characteristic. If the overall confirmedinconsistency score of the entire application exceeds a threshold value,a downstream machine-learning component uses the inconsistencyinformation to train the application to better respond to future inputdatasets that share certain characteristics with the current inputdataset. In some embodiments, a human developer or computerized designapplication would also attempt to determine whether the semanticinconsistencies occurred as a function of a correctable design error inthe application.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a cloud computing environment according to an embodimentof the present invention.

FIG. 2 depicts abstraction model layers according to an embodiment ofthe present invention.

FIG. 3 shows the structure of a computer system and computer programcode that may be used to implement a method for improved semanticconsistency of explanations in explainable artificial intelligenceapplications in accordance with embodiments of the present invention.

FIG. 4 shows a high-level architecture of an XAI-based artificiallyintelligent software application with enhancements for providingimproved semantic consistency of explanations, in accordance withembodiments of the present invention.

FIG. 5 is a flow chart that illustrates steps of a method for improvedsemantic consistency of explanations produced by explainable artificialintelligence applications in accordance with embodiments of the presentinvention.

DETAILED DESCRIPTION

Explainable Artificial Intelligence (XAI) refers to applications ofartificial intelligence technology (AI) that explain how they arrived ata particular decision. The format of these explanations isimplementation-dependent, including combinations of natural-languagetext, numeric codes, Boolean expressions, or other mechanisms forcommunicating the reasons why an XAI application made a certain decisionor produced certain output. XAI technology may be incorporated into manytypes of AI applications, such as expert systems and other types ofself-learning.

XAI augments “black box” AI and machine-learning technologies in whicheven an application's designer cannot be sure why the applicationarrived at a particular decision. An explainable AI application canenumerate inferences upon which such decisions are based, informingusers, designers, administrators, or downstream systems how, when, andhow well a particular AI model works. This information can be used todecide whether an application requires further training or whether itwould be practical to migrate a particular AI application, such as atrained deep-learning neural network, to a different field.

When an XAI application comprises an ensemble of serially connectedmodels, explanations produced by each model are most useful if they areformally and logically consistent. This can be especially challengingwhen models migrated from different applications produce discrepanciesor when different designers have modeled the same applicationdifferently, or for a different purpose. In such cases, the explanationsor output produced by these models may not be consistent.

For example, a multi-lingual AI-based conversational system may seem toproduce different results when certain types of otherwise-identicalinput datasets are entered in different languages. Similarly, certaininputs may cause a fraud-detection system to reach conclusions that,when considered within the context of national averages, appears to bestatistical outliers. In such cases, if explanations produced bydifferent modules of a known XAI system are semantically inconsistent,those explanations must be considered unreliable.

Even an application that is presumed to have been adequately trained maysuffer from subtle, infrequent inferential errors and inconsistenciesthat were not detected during the application's machine-learningsessions or other known training and verification mechanisms. XAIsystems are especially vulnerable to these types of problems whensemantic inconsistencies between two models' explanations make it hardto know when or why the models' artificially intelligent inferences haveproduced contradictory or irrelevant results.

When this happens, a system that is known to have been producing goodresults may unexpectedly generate unexpected outcome when fed aparticular input dataset. Known explainability functions may in suchcases produce contradictory or inconsistent explanations, making itdifficult to tell whether the unexpected outcome is erroneous or isexplainable as an unusual, but correct, result. This problem isespecially troublesome if there is a possibility that an offendingdataset was generated by a hostile party in order to defraud orotherwise deceive the XAI system.

Embodiments of the present invention improve known XAI technology byproviding a mechanism for quantifying semantic consistency amongexplanations produced by modules of an XAI-based application whenprocessing certain inputs, and using the resulting analysis to pinpointthose modules responsible for the inconsistency. This allows developers,automated application-management tools, or downstream systems to betterdetermine whether the seemingly anomalous results should be trusted and,if necessary, to then take remedial action, such as further training orredesigning some or all of the XAI modules.

These improvements to known XAI technology improve XAI applications andsystems by providing more robust operation, the ability to handle abroader range of input, and greater overall reliability. Theseimprovements also make XAI-based ensembles easier to maintain and debugbecause they provide useful debugging information without requiring, asdo known implementations, analysis of each individual module in order toidentify the source of a possible error.

It is to be understood that although this disclosure includes a detaileddescription on cloud computing, implementation of the teachings recitedherein are not limited to a cloud computing environment. Rather,embodiments of the present invention are capable of being implemented inconjunction with any other type of computing environment now known orlater developed.

Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g., networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services) that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model may includeat least five characteristics, at least three service models, and atleast four deployment models.

Characteristics are as Follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but may be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported, providing transparency for both theprovider and consumer of the utilized service.

Service Models are as Follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based e-mail).The consumer does not manage or control the underlying cloudinfrastructure including network, servers, operating systems, storage,or even individual application capabilities, with the possible exceptionof limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as Follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure that includes anetwork of interconnected nodes.

Referring now to FIG. 1 , illustrative cloud computing environment 50 isdepicted. As shown, cloud computing environment 50 includes one or morecloud computing nodes 10 with which local computing devices used bycloud consumers, such as, for example, personal digital assistant (PDA)or cellular telephone 54A, desktop computer 54B, laptop computer 54C,and/or automobile computer system 54N may communicate. Nodes 10 maycommunicate with one another. They may be grouped (not shown) physicallyor virtually, in one or more networks, such as Private, Community,Public, or Hybrid clouds as described hereinabove, or a combinationthereof. This allows cloud computing environment 50 to offerinfrastructure, platforms and/or software as services for which a cloudconsumer does not need to maintain resources on a local computingdevice. It is understood that the types of computing devices 54A-N shownin FIG. 1 are intended to be illustrative only and that computing nodes10 and cloud computing environment 50 can communicate with any type ofcomputerized device over any type of network and/or network addressableconnection (e.g., using a web browser).

Referring now to FIG. 2 , a set of functional abstraction layersprovided by cloud computing environment 50 (FIG. 1 ) is shown. It shouldbe understood in advance that the components, layers, and functionsshown in FIG. 2 are intended to be illustrative only and embodiments ofthe invention are not limited thereto. As depicted, the following layersand corresponding functions are provided:

Hardware and software layer 60 includes hardware and softwarecomponents. Examples of hardware components include: mainframes 61; RISC(Reduced Instruction Set Computer) architecture based servers 62;servers 63; blade servers 64; storage devices 65; and networks andnetworking components 66. In some embodiments, software componentsinclude network application server software 67 and database software 68.

Virtualization layer 70 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers71; virtual storage 72; virtual networks 73, including virtual privatenetworks; virtual applications and operating systems 74; and virtualclients 75.

In one example, management layer 80 may provide the functions describedbelow. Resource provisioning 81 provides dynamic procurement ofcomputing resources and other resources that are utilized to performtasks within the cloud computing environment. Metering and Pricing 82provide cost tracking as resources are utilized within the cloudcomputing environment, and billing or invoicing for consumption of theseresources. In one example, these resources may include applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal 83 provides access to the cloud computing environment forconsumers and system administrators. Service level management 84provides cloud computing resource allocation and management such thatrequired service levels are met. Service Level Agreement (SLA) planningand fulfillment 85 provide pre-arrangement for, and procurement of,cloud computing resources for which a future requirement is anticipatedin accordance with an SLA.

Workloads layer 90 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation 91; software development and lifecycle management 92; virtualclassroom education delivery 93; data analytics processing 94;transaction processing 95; and orchestration of complex methods,systems, and computer program products for improved semantic consistencyof machine-learning explanations 96.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a computer, or other programmable data processing apparatusto produce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks. These computerreadable program instructions may also be stored in a computer readablestorage medium that can direct a computer, a programmable dataprocessing apparatus, and/or other devices to function in a particularmanner, such that the computer readable storage medium havinginstructions stored therein comprises an article of manufactureincluding instructions which implement aspects of the function/actspecified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be accomplished as one step, executed concurrently,substantially concurrently, in a partially or wholly temporallyoverlapping manner, or the blocks may sometimes be executed in thereverse order, depending upon the functionality involved. It will alsobe noted that each block of the block diagrams and/or flowchartillustration, and combinations of blocks in the block diagrams and/orflowchart illustration, can be implemented by special purposehardware-based systems that perform the specified functions or acts orcarry out combinations of special purpose hardware and computerinstructions.

FIG. 3 shows a structure of a computer system and computer program codethat may be used to implement a method for improved semantic consistencyof explanations in explainable artificial intelligence applications inaccordance with embodiments of the present invention. FIG. 3 refers toobjects 301-315.

In FIG. 3 , computer system 301 comprises a processor 303 coupledthrough one or more I/O Interfaces 309 to one or more hardware datastorage devices 311 and one or more I/O devices 313 and 315.

Hardware data storage devices 311 may include, but are not limited to,magnetic tape drives, fixed or removable hard disks, optical discs,storage-equipped mobile devices, and solid-state random-access orread-only storage devices. I/O devices may comprise, but are not limitedto: input devices 313, such as keyboards, scanners, handheldtelecommunications devices, touch-sensitive displays, tablets, biometricreaders, joysticks, trackballs, or computer mice; and output devices315, which may comprise, but are not limited to printers, plotters,tablets, mobile telephones, displays, or sound-producing devices. Datastorage devices 311, input devices 313, and output devices 315 may belocated either locally or at remote sites from which they are connectedto I/O Interface 309 through a network interface.

Processor 303 may also be connected to one or more memory devices 305,which may include, but are not limited to, Dynamic RAM (DRAM), StaticRAM (SRAM), Programmable Read-Only Memory (PROM), Field-ProgrammableGate Arrays (FPGA), Secure Digital memory cards, SIM cards, or othertypes of memory devices.

At least one memory device 305 contains stored computer program code307, which is a computer program that comprises computer-executableinstructions. The stored computer program code includes a program thatimplements a method for improved semantic consistency of explanations inexplainable artificial intelligence applications in accordance withembodiments of the present invention, and may implement otherembodiments described in this specification, including the methodsillustrated in FIGS. 1-5 . The data storage devices 311 may store thecomputer program code 307. Computer program code 307 stored in thestorage devices 311 is configured to be executed by processor 303 viathe memory devices 305. Processor 303 executes the stored computerprogram code 307.

In some embodiments, rather than being stored and accessed from a harddrive, optical disc or other writeable, rewriteable, or removablehardware data-storage device 311, stored computer program code 307 maybe stored on a static, nonremovable, read-only storage medium such as aRead-Only Memory (ROM) device 305, or may be accessed by processor 303directly from such a static, nonremovable, read-only medium 305.Similarly, in some embodiments, stored computer program code 307 may bestored as computer-readable firmware, or may be accessed by processor303 directly from such firmware, rather than from a more dynamic orremovable hardware data-storage device 311, such as a hard drive oroptical disc.

Thus the present invention discloses a process for supporting computerinfrastructure, integrating, hosting, maintaining, and deployingcomputer-readable code into the computer system 301, wherein the code incombination with the computer system 301 is capable of performing amethod for improved semantic consistency of explanations in explainableartificial intelligence applications.

Any of the components of the present invention could be created,integrated, hosted, maintained, deployed, managed, serviced, supported,etc. by a service provider who offers to facilitate a method forimproved semantic consistency of explanations in explainable artificialintelligence applications. Thus the present invention discloses aprocess for deploying or integrating computing infrastructure,comprising integrating computer-readable code into the computer system301, wherein the code in combination with the computer system 301 iscapable of performing a method for improved semantic consistency ofexplanations in explainable artificial intelligence applications.

One or more data storage devices 311 (or one or more additional memorydevices not shown in FIG. 3 ) may be used as a computer-readablehardware storage device having a computer-readable program embodiedtherein and/or having other data stored therein, wherein thecomputer-readable program comprises stored computer program code 307.Generally, a computer program product (or, alternatively, an article ofmanufacture) of computer system 301 may comprise the computer-readablehardware storage device.

In embodiments that comprise components of a networked computinginfrastructure, a cloud-computing environment, a client-serverarchitecture, or other types of distributed platforms, functionality ofthe present invention may be implemented solely on a client or userdevice, may be implemented solely on a remote server or as a service ofa cloud-computing platform, or may be split between local and remotecomponents.

While it is understood that program code 307 for a method for improvedsemantic consistency of explanations in explainable artificialintelligence applications may be deployed by manually loading theprogram code 307 directly into client, server, and proxy computers (notshown) by loading the program code 307 into a computer-readable storagemedium (e.g., computer data storage device 311), program code 307 mayalso be automatically or semi-automatically deployed into computersystem 301 by sending program code 307 to a central server (e.g.,computer system 301) or to a group of central servers. Program code 307may then be downloaded into client computers (not shown) that willexecute program code 307.

Alternatively, program code 307 may be sent directly to the clientcomputer via e-mail. Program code 307 may then either be detached to adirectory on the client computer or loaded into a directory on theclient computer by an e-mail option that selects a program that detachesprogram code 307 into the directory.

Another alternative is to send program code 307 directly to a directoryon the client computer hard drive. If proxy servers are configured, theprocess selects the proxy server code, determines on which computers toplace the proxy servers' code, transmits the proxy server code, and theninstalls the proxy server code on the proxy computer. Program code 307is then transmitted to the proxy server and stored on the proxy server.

In one embodiment, program code 307 for a method for improved semanticconsistency of explanations in explainable artificial intelligenceapplications is integrated into a client, server and network environmentby providing for program code 307 to coexist with software applications(not shown), operating systems (not shown) and network operating systemssoftware (not shown) and then installing program code 307 on the clientsand servers in the environment where program code 307 will function.

The first step of the aforementioned integration of code included inprogram code 307 is to identify any software on the clients and servers,including the network operating system (not shown), where program code307 will be deployed that are required by program code 307 or that workin conjunction with program code 307. This identified software includesthe network operating system, where the network operating systemcomprises software that enhances a basic operating system by addingnetworking features. Next, the software applications and version numbersare identified and compared to a list of software applications andcorrect version numbers that have been tested to work with program code307. A software application that is missing or that does not match acorrect version number is upgraded to the correct version.

A program instruction that passes parameters from program code 307 to asoftware application is checked to ensure that the instruction'sparameter list matches a parameter list required by the program code307. Conversely, a parameter passed by the software application toprogram code 307 is checked to ensure that the parameter matches aparameter required by program code 307. The client and server operatingsystems, including the network operating systems, are identified andcompared to a list of operating systems, version numbers, and networksoftware programs that have been tested to work with program code 307.An operating system, version number, or network software program thatdoes not match an entry of the list of tested operating systems andversion numbers is upgraded to the listed level on the client computersand upgraded to the listed level on the server computers.

After ensuring that the software, where program code 307 is to bedeployed, is at a correct version level that has been tested to workwith program code 307, the integration is completed by installingprogram code 307 on the clients and servers.

Embodiments of the present invention may be implemented as a methodperformed by a processor of a computer system, as a computer programproduct, as a computer system, or as a processor-performed process orservice for supporting computer infrastructure.

FIG. 4 shows a high-level architecture of an XAI-based artificiallyintelligent software application 4000 and enhancements 425-435 forproviding improved semantic consistency of explanations, in accordancewith embodiments of the present invention. FIG. 4 shows items 4000 and400-455. Items that have subscripted labels in FIG. 4 are equivalent inform and function to items labeled in the below text with parentheticallabels. For example, label M(N) in the text below identifies thegraphical object labeled M_(N) in the FIG. 4 .

XAI-compliant application 4000 is comprised of N artificiallyintelligent modules M(1)-M(N) 410 a-410 d, each of which produces acorresponding output OUT(1)-OUT(N) 410 a-410 d and an explanationE(1)-E(N) 415 a-415 d of each corresponding output. Modules M(1)-M(N)410 a-410 d could, for example, be sequentially linked expert systems orcould each be layers of a single deep-learning AI application, aconvolutional neural network (CNN), or a long-term recurrent convolutionnetwork (LTRC) 4000, where each module implements a cognitive model thatgenerates output forwarded as input to the next module/model in theapplication 4000. For example, module M(2) 410 b generates output OUT(2)405 b, which becomes input for the next module M(3) 410 c, and generatesan explanation E(2) 415 b, which provides an explanation of outputOUT(2) 405 b, such as a Boolean expression, data structure, or text thatidentifies context, logic, or conditions from which decisions underlyingoutput OUT(2) 405 b were inferred.

In the embodiment of FIG. 4 , Application 4000 receives an input dataset400, which is forwarded as input to the first XAI-based module M(1) 410a. The artificially intelligent model embodied by module M(1) 410 a,based on receiving input 400, forwards output OUT(1) 405 a to the nextmodule in application 4000, module M(2) 410 b. Module M(2) 410 b thengenerates output OUT(2) 405 b, which is forwarded to the next moduleM(3) 410 c, and so forth. This sequential process ends when the finalmodule of application 4000, module M(N) 405 d, generates output OUT(N)405 d, which becomes the overall output 420 of the entire application4000.

In certain implementations, the elements of input data set 400 arepropagated as input to all modules M(1)-M(N) 410 a-410 d. In such caseseach module 410 b-410 d generates output as a function of both theoutput of an immediately preceding module and of the original values ofelements of input dataset 400.

Each module M(1)-M(N) 410 a-410 d generates an explanation E(1)-E(N) 415a-415 d that is forwarded to a semantic-consistency profiler module 425(or profiler 425). Profiler 425 determines, by means of known methods ofartificial intelligence, whether any inconsistencies between pairs ofexplanations E(1)-E(N) 415 a-415 d should be ignored in light of knownattributes of system input 400 and boundary conditions or otherconstraints of the modules that generate the pair of seeminglyinconsistent explanations. If so, those explanations are eliminated fromfurther processing by Explainability Analyzer 430.

The remaining explanations of explanations E(1)-E(N) 415 a-415 d areforwarded to explainability analyzer 430, which generates an“Inconsistency Vector” that quantifies the degree of inconsistencybetween each pair of the remaining explanations.

Inconsistency log generator (435 (or log generator 435) creates aninconsistency log of the semantic inconsistencies identified byexplainability analyzer 430. The log comprises a record of eachinconsistency, identifying the pair of modules and the input dataset 400that generated the inconsistent explanations. In certain embodiments,the log also identifies the output of each module that generated aninconsistent explanation.

The log is fed to the training system 440 that trains XAI application4000. Training system 440 may implement any mechanism known for trainingan artificially intelligent application, such as the training componentsof a machine-learning application. Training system 440, using AItraining methods known in the art, incorporates the log information intoa machine-learning training corpus or other type of formalized trainingdata 445. This training data 445 is used to further train application4000 to recognize, report, or take action to mitigate inconsistenciessimilar to those identified by the inconsistency log. In someembodiments, the updated corpus may be resubmitted to application 4000as a new input dataset 400, possibly after application 4000 has learnedhow to properly respond to the new input dataset 400 or has beendeliberately redesigned to know better how to respond to the new inputdataset 400.

FIG. 5 is a flow chart that illustrates steps of a method for improvedsemantic consistency of explanations produced by explainable artificialintelligence applications in accordance with embodiments of the presentinvention. FIG. 5 shows steps 500-570.

In step 500, an enhanced explainable artificially intelligent (XAI)system receives a set of N explanations E(1)-E(N) 415 a-415 d frommodules M(1)-M(N) 410 a-410 d of an artificially intelligent application4000. As explained in FIG. 4 , modules 410 a-410 d are organizedsequentially in application 4000 such that an i^(th) module M(i) in thesequence generates an i^(th) output OUT(i) that is forwarded as input toa next (i+1^(st)) module in the sequence, and where the i^(th) moduleM(i) also generates an i^(th) explanation E(i) of the reasoning uponwhich the i^(th) output is based.

In certain embodiments, each module M(1)-M(N) 410 a-410 d generates acorresponding output OUT(1)-OUT(N) 405 a-405 d and a correspondingexplanation E(1)-E(N) 415 a-415 d as a further function of receiving theinput dataset 400 that had been submitted to application 4000 from anextrinsic source. In other words, each module M(1)-M(N) 410 a-410 dgenerates an output and an explanation based on both the value of eachelement of input dataset 400 and on the value of each element of theoutput, if any, produced by a module that is the immediately precedingmodule in the sequence of modules that make up application 4000.

The first module in the sequence (M(1) 410 a) receives as input only theinput dataset 400 that has been submitted to application 4000 from anextrinsic source. Output OUT(N) 405 d, generated by the final module inthe sequence (M(N) 410 d), is forwarded by application 4000 as systemoutput to downstream systems or users.

Each explanation may be formatted in any manner known in the art.Examples and embodiments presented in this document compriseexplanations E(1)-E(N) 415 a-415 d that each contain TRUE or FALSEBoolean expressions. Here, each element of an i^(th) explanation is alogical expression that represents or provides context or otherexplanation of an i^(th) element of a corresponding output. For example,if a module M(j) produces an output O(j) and an explanation E(j), thefourth element of output O(j) may result from processing, by moduleM(j), a fourth element of an input dataset 400 and may be explained by aBoolean expression of the fourth element of explanation E(j). In otherwords, if input dataset 400 contains a set or array of elements, amodule of application 4000 produces an output that contains a similarset or array of elements, at least some of which correspond torespective elements of input dataset 400. That module also produces anexplanation that contains an array of elements that each corresponds toa respective element of the module's output.

This convention is presented for pedagogical reasons and should not beconstrued to limit embodiments of the present invention toBoolean-expression explanations. The present invention is flexibleenough to accommodate any sort of explanation format that is preferredby an implementer, so long as that format does not prevent an embodimentfrom identifying or quantifying inter-explanation semanticinconsistencies, in a manner similar to those described in subsequentsteps of FIG. 5 .

In step 510, the system identifies semantic inconsistencies betweencorresponding elements of each possible pairing of explanationsE(1)-E(N) 415 a-415 d received in step 510. For example, in this step,the system might compare elements of explanation E(1) 415 a tocorresponding elements of explanation E(3) 415 c, comparing the firstelement of E(1) 415 a to the first element of E(3) 415 c, the secondelement of E(1) 415 a to the second element of E(3) 415 c, and so forth,repeating the entire procedure to compare every other combination of twoexplanations produced by modules of application 4000.

In examples and embodiments described herein, two explanation elementsare deemed to be semantically inconsistent if the two Booleanexpressions embodied by the two elements are logically inconsistent oridentify disjoint, non-intersecting sets of results. For example,consider an application 4000 that infers or predicts characteristics oftransactions performed by university students and teachers. In responseto the submission of a 40-element input dataset 400 to application 4000,two modules of application 4000, M(k) and M(m), respectively generate40-element outputs O(k) and O(m) and 40-element explanations E(k) andE(m). Each element of these explanations is a Boolean expression fromwhich may be inferred explanations of the reasoning used by acorresponding module to derive a corresponding output element.

In this example, each element of E(k) consists of the Booleanexpression:E(k)=(TransactionSize≥Th)Λ(UserProfile=Teacher)

and each element of E(m) consists of the Boolean expression:E(m)=(TransactionSize≥Th)Λ(UserProfile=Student)

where Th is a threshold value of a transaction size.

This means that an element of E(k) is a Boolean expression thatevaluates to a value of TRUE if a corresponding transaction is performedby a Teacher and a transaction size greater than Th. Similarly, E(m) isa Boolean expression that evaluates to a value of TRUE if acorresponding transaction is performed by a Student and that thetransaction is larger than Th. These expressions are evaluated in asimilar way for each element of output O(k) or O(m), respectively. Insome embodiments, certain modules may produce output that does notcontain elements that correspond to every element of an input dataset.For example, a final module of an application that has received a10-element input dataset may generate only 8 output elements. But evenin such a case, an implementer may choose to configure an embodimentsuch that a module's output and explanation still contain elements thatcorrespond to every element of the output.

In the current example, input dataset 400 consists of elements that eachreference a particular transaction, identified by an element of inputdataset 400, that has been performed by students, teachers, orcombinations of students and teachers. Module M(k) identifiestransactions of a certain size in which at least one transacting partyis a teacher. Module M(m) identifies transactions of a certain size inwhich at least one transacting party is a student.

For example, if the sixth element of input dataset 400 identifies atransaction between two teachers that is larger than threshold Th, thesixth element of explanation E(k) evaluates to a value of TRUE and thesixth element of explanation E(m) evaluates to a value of FALSE. Thesevalues explain why the sixth element of output O(k) identifies atransaction that satisfies conditions of module M(m) and why the sixthelement of output P(m) is an empty set. Similarly, if the third elementof input dataset 400 identifies a transaction between two students thatis larger than threshold Th, the third element of explanation E(k) isFALSE (explaining why the third element of output O(k) is empty) and thesixth element of explanation E(m) is TRUE. If the seventh element ofinput dataset 400 identifies a transaction between two students that issmaller than threshold Th, the seventh elements of explanation E(k) andexplanation E(m) are both FALSE. Finally, if the eighth element of inputdataset 400 identifies a transaction between a teacher and a studentthat is larger than threshold Th, the eighth elements of explanationE(k) and explanation E(m) are both TRUE.

A potential semantic inconsistency between an element of explanationE(k) and a corresponding element of explanation E(m) occurs whencorresponding elements of two outputs are disjoint or, equivalently,when corresponding elements of the two explanations are inconsistent. Inthe running example, a possible inconsistency thus occurs when anelement of input dataset 400 identifies a transaction that falls belowthreshold Th, resulting in disjoint outputs of modules M(k) and M(m) andFALSE values of explanations M(k) and M(m).

As will be explained below, some potential inconsistencies identified inthis step, rather than being true inconsistencies, are merely expectedresults of submitting a particular input dataset. Existing methods ofgenerating and interpreting XAI explanations cannot distinguish betweena true training insufficiency of an application 4000 and correct, butseemingly anomalous, input-dependent result. However, embodiments of thepresent invention improve upon current XAI technology by providing a wayto determine when an inconsistency between explanation elementsindicates that an application 4000 has produced a true error.

At the conclusion of step 510, the system will have identified potentialsemantic inconsistencies between corresponding elements of pairs of twoexplanations E(1)-E(N) 415 a-415 d. In certain embodiments, the systemwill have identified potential semantic inconsistencies between allcorresponding elements of certain pairs of explanations E(1)-E(N) 415a-415 d, or will have identified potential semantic inconsistenciesbetween corresponding elements every possible combination of twoexplanations E(1)-E(N) 415 a-415 d. If desired by an implementer, theseinconsistencies will have been identified with sufficient granularity todetermine exactly how many and which elements of an explanation pair arepotentially inconsistent.

In step 520, the system assigns an inconsistency score to each pair ofexplanations that were compared in step 510. In some embodiments, aninconsistency score is assigned in this step to each possiblecombination of two explanations.

Inconsistency scores may be derived by any means known in the art, andmay be derived by means of known or novel statistical, probabilistic, orcognitive method, including artificially intelligent methods basedtechnologies like semantic analytics, machine learning, or textanalytics. In all cases, an inconsistency score is a quantitativemeasure of a degree of semantic inconsistency between two explanations415 a-415 d of application 4000.

Embodiments and examples described in this document describe a type ofinconsistency score that comprises a vector or matrix I in which eachelement describes a degree of inconsistency between one pair ofexplanations produced by application 4000. These embodiments andexamples should not be construed to limit the present invention to suchscores. The present invention is flexible enough to accommodateembodiments that represent a degree of inconsistency between twoexplanations in any manner desired by an implementer.

Embodiments that adopt the aforementioned vector or matrixrepresentation I of an inconsistency score contain elements that eachquantify a degree of inconsistency between a pair of explanations E(i)and E(j) of application 4000, expressed as:I(i,j)=# elements for which Boolean satisfiability of E(i) andE(j)contradict

total #elements

For example, if explanations E(i) and E(j) each contain 20 elements, andthe system in step 510, determined that four of those elements arepotentially semantically inconsistent, then I(i,j) represents a degreeof inconsistency between explanations E(i) and E(j) as:I(i,j)=4=0.2

20

In some embodiments, the total number of elements is defined as thenumber of elements in input dataset 400 or in system output 420 (theoutput of the final module 410 d of application 4000). In otherembodiments, the total number of elements, if preferred by animplementer, is defined as the number of elements contained inexplanation E(i) or in explanation E(j) or as the number of pairs ofelements in E(i) and E(j) that were compared in step 510.

At the conclusion of step 520, the system will have developed aninconsistency score of application 4000 that quantifies degrees ofinconsistency between pairs of explanations produced by application 4000in response to receiving a particular input dataset 400. Each degree ofinconsistency in the score uniquely identifies one distinct degree ofinconsistency between one distinct pair of explanations and no distinctpair of explanations is scored by more than one element of theinconsistency score.

In step 530, the system identifies, through inference or analysis,previously identified inconsistencies that are likely to be falsepositives. The system then revises the previously derived degrees ofinconsistency such that each false-positive inconsistency is set tozero.

For example, in the above exemplary student-teacher transactionembodiment, an I(k,m) element of an inconsistency vector I may evaluateto a value of 1.0 if all 40 elements of explanations E(k) and E(m) arefound to be semantically inconsistent. This may occur if none of the 40elements of E(k) and E(m) overlap.

Such a result could occur if module M(k) or M(m) produces an explanationE(k) or E(m) that does not correctly explain how module M(k) or M(m)derived its respective output O(k) or O(m) after the submission ofdataset 400 to application 4000 and could occur if module M(k) or M(m)does not derive an expected output O(k) or O(m) after the submission ofdataset 400 to application 4000.

However, it is also possible that explanations E(k) and E(m) and outputsO(k) and O(m) are all correct and that the seemingly inconsistentexplanations appear to be inconsistent because of a characteristic of anoutlier, or unexpected, input dataset 400. For example, if the inputdataset 400 identifies no transaction that exceeds minimum transactionsize Th, then application 400's disjoint outputs O(k) and O(m) andseemingly inconsistent explanations E(k) and E(m) would have correctlyindicated a statistically unlikely characteristic of the input dataset400, rather than indicating a nonexistent error in the reasoning ofmodule M(k) or M(m).

The same would be true if input dataset 400 references no transactionsbetween students and teachers. In this case, the output set of moduleM(k) would reference only transactions performed by teachers but notstudents and the output set of module M(m) would reference onlytransactions performed by students but not teachers. The two outputsO(m) and O(k) would thus be disjoint, and explanations M(k) and M(m)would have been flagged as potentially being inconsistent. However,these results would not indicate a training or design error in moduleM(k) or M(m) because the disjoint output and FALSE Boolean values of theexplanations would have been correct, if unexpected, results of anunexpected characteristic of input dataset 400.

In such a case, element I(k,m) would be reset by the system in step 530to a value of 0, indicating that no semantic inconsistency has beendetected between explanations E(k) and E(m). All other elements of theinconsistency vector are considered and revised as necessary in thisstep.

The system may determine when a possible inconsistency between twoexplanations, derived in step 510 and 520 is a false positive throughany means known in the art. For example, the system may, throughartificial intelligence, cognitive analytics, statistical orprobabilistic analysis, or machine-learning technology, have beentrained to infer semantic meaning to correlations, dependencies, andother relationships between input-dataset values and module outputs. Inthe teacher-student example, such an embodiment could identify a falsepositive by inferring that the Boolean expressions produce TRUE resultsonly when certain input conditions exist, such as the existence of atleast one input transaction between at least one teacher and one studentthat exceeds a certain size. These types of inferences and cognitivereasoning can be much more complex in real-world implementations, butare within the current state of the art of inferential computingtechnologies.

Furthermore, each time the system guesses that a previously identifiedinconsistency is actually a false positive, that guess can be confirmedthrough cognitive methods, such as by submitting the guess and itsultimate resolution to a machine-learning module that incorporates thatdata into a training corpus. Even incorrect identifications of afalse-positive inconsistency can thus improve the ability of the systemto correctly identify false-positives that occur with future inputdatasets 400, but which, despite being possibly unexpected, fall withinthe boundary constraints of application 4000's design.

The filtering of step 530 is an improvement to current XAI technology,which cannot identify when explanations produced by modules of an XAIapplication are semantically inconsistent in order to identify potentialtraining or design errors in one or more modules of the application.Even if this was not true, known XAI applications cannot identify whichidentified inconsistencies are actually false positive identificationsthat are unexpected artifacts of a characteristic of a particular inputdataset. These improvements allow a training module, such as a trainingmodule of a machine-learning application, to more accurately andcompletely train the XAI application, even when it had been previouslypresumed that the application was already fully trained.

In step 540, the system determines whether the inconstancy score derivedin steep 520 indicates a likelihood of a training or logic error inapplication 4000. The system may interpret the contents of theinconsistency score, in order to infer such a likelihood, by any meansknown in the art. For example, an implementer may identify aninconsistency threshold value of 0.5, such that of any degree ofinconsistency between a pair of modules would result in a determinationthat application 4000 is likely to suffer from a training insufficiencyor design error; and that one or both of the pair of modules is mostlikely to be a local source of the insufficiency or error.

In other embodiments, the system in this step might infer a likelihoodof a training or logic error in application 4000 if a total number ofelements of an inconsistency vector exceed a threshold degree ofinconsistency. For example, if a threshold degree of inconsistency ispredefined as a value 0.2 and a threshold number of inconsistencies ispredefined as a value of 5 when an input dataset contains 40 elements,then the system in step 540 would determine whether more than fiveelements of inconsistency vector I indicate degrees of inconsistency,between a pair of modules, greater than 0.2.

Other embodiments may comprise different or additional methods ofidentifying conditions necessary to identifying unacceptableinconsistencies, as desired by an implementer. In all cases, if thepreviously derived inconsistency scores satisfy those conditions, themethod of FIG. 5 continues with steps 550-570. If the previously derivedinconsistency scores do not satisfy the conditions necessary to indicatea likelihood that application 4000 suffers from a training or logicerror that results in undesirable system output 420 when the currentdataset 400 is submitted to application 4000, then the method of FIG. 5ends. In this case, the method of FIG. 5 resumes with step 500 the nexttime that application 4000 produces explanations in response toreceiving another set of input data.

In step 550, the system generates one or more log entries or otherrecord of the results of previous steps of FIG. 5 . This record maycomprise any information deemed relevant by an implementer, but in allembodiments must include at least an identification of at least one pairof modules that have been determined to be sources of a confirmedsemantic inconsistency between two explanations, an identification ofthe input dataset 400 that, when submitted to application 4000, resultedin the confirmed semantic inconsistency. Some embodiments may also addcontextual or complementary information, such as an identification ofthe outputs of the two modules, an identification of the semanticallyinconsistent explanations, a record of the inconsistency scores ordegrees of inconsistency that resulted in the performance of steps550-570, or descriptions of design logic, requirements, prior trainingdata, or other data from which may be identified a function orcharacteristic of application 4000.

In step 560, the system forwards the log entries or other record todownstream systems, to a machine-learning training application, or to ahuman administrator capable of training application 4000 to betterhandle input similar to that of input dataset 4000 or to address apossible design flaw suggested by the inconsistency results. Forexample, in the exemplary student-teacher embodiment, the forwardedrecord may contain information that can be used by a machine-learningtraining application to train application 4000 to better handle futureinput that specifies the same combination of transactions that resultedin semantic inconsistencies the last time that such transactions weresubmitted to application 4000 as the most recent input dataset 400.

In step 570, a machine-learning training component of the systemincorporates the information forwarded to the component in step 560 intoa machine-learning corpus that is then used to further train application4000.

Each iteration of the method of FIG. 5 is performed whenever it isdetermined that application 4000, in response to receiving a specificinput dataset 400, has produced unexpected output. In this manner,iterative identifications of semantic inconsistencies in theexplanations produced by XAI application 4000 facilitate more refinedtraining of application 4000, allowing application 4000 to better handleunusual types in input data.

In some embodiments, the system may take additional steps aimed atfurther improving the accuracy of application 4000 or the usefulness ofexplanations produced by application 4000.

For example, the system, based on results of inconsistency evaluationsmade by steps of the method of FIG. 5 , may determine thatfalse-positive explanations have over time been related to aninconsistent use of terminology in explanations produced by application4000. This could happen when modules of application 4000 were developedby different developers at different times, or were repurposed fromartificially intelligent applications originally developed for differentpurposes. In such cases, the system may infer, through cognitive orother types of inferential technologies, that false-positiveexplanations may have been made more likely by formal inconsistenciesamong explanations. The system would then advise an administrator ordesigner to revise application 4000 to produce more consistentexplanations, or might automatically train application 4000, through amethod of machine-learning, to produce explanations that are formallyconsistent. In other cases, the system might direct application 4000, ora downstream application, to ensure that explanations generated byapplication 4000 conform to a certain ontology or ontologies that ensurethe explanations comprise consistent terminology, structures, orformats.

Examples and embodiments of the present invention described in thisdocument have been presented for illustrative purposes. They should notbe construed to be exhaustive nor to limit embodiments of the presentinvention to the examples and embodiments described here. Many othermodifications and variations of the present invention that do not departfrom the scope and spirit of these examples and embodiments will beapparent to those possessed of ordinary skill in the art. Theterminology used in this document was chosen to best explain theprinciples underlying these examples and embodiments, in order toillustrate practical applications and technical improvements of thepresent invention over known technologies and products, and to enablereaders of ordinary skill in the art to better understand the examplesand embodiments disclosed here.

What is claimed is:
 1. An explainable artificially intelligent (XAI)system comprising: a processor; a memory coupled to the processor; and acomputer-readable hardware storage device coupled to the processor, thestorage device containing program code configured to be run by theprocessor to implement a method comprising: in response to a submissionof an input dataset to an artificially intelligent application,receiving, by the processor, explanations from modules of theapplication, wherein the modules are configured within the applicationin a serial sequence in which each module, upon receiving the inputdataset and any input generated by an immediately preceding module ofthe serial sequence, generates output that is forwarded as input to anext module, if any, in the sequence, and wherein each explanation ofthe received explanations explains an artificially intelligent reasoningby which a corresponding output is generated by a corresponding moduleof the application; and determining that at least two of the receivedexplanations are semantically inconsistent.
 2. The system of claim 1,wherein the determining comprises: identifying one or more potentiallyinconsistent pairs of explanations, of the received explanations, byperforming a semantic analysis upon each explanation of the receivedexplanations; and filtering out a false-positive pair of the potentiallyinconsistent pairs by using an artificially intelligent method ofcognitive computing to determine that inconsistencies between twoexplanations of the false-positive pair are an expected consequence of acharacteristic of the input dataset.
 3. The system of claim 2, whereinthe more accurate response to inputs that share the characteristic withthe input dataset comprises more accurately identifying thefalse-positive pair.
 4. The system of claim 1, wherein the determiningcomprises: quantifying a degree of inconsistency between the twosemantically inconsistent explanations as a function of a total numberof elements of the input dataset that are associated with a Booleaninconsistency between a first explanation of the two semanticallyinconsistent explanations and a second explanation of the twosemantically inconsistent explanations; and identifying that the degreeof inconsistency exceeds a predefined threshold level.
 5. The system ofclaim 1, wherein the determining comprises: determining an existence ofa systemic error in one or both of two modules of the application inresponse to a determination that a corresponding element of aninconsistency vector exceeds a predefined threshold inconsistency value,wherein each element of the inconsistency vector identifies a relativenumber of elements of the input dataset that are associated with aBoolean inconsistency between the explanations generated by the twomodules.
 6. The system of claim 5, wherein the method comprises:recording, by the processor, a log entry in an inconsistency log,wherein the log entry identifies the modules corresponding to the twosemantically inconsistent explanations and the input dataset;forwarding, by the processor, the log to a machine-learning trainingcomponent; and directing, by the processor, the machine-learningtraining component to incorporate contents of the log entry into amachine-learning corpus to be submitted to the application during asubsequent machine-learning training session.
 7. The system of claim 1,wherein each explanation is selected from the group consisting of: aBoolean expression, natural-language text, and a string of alphanumericcharacters.
 8. The system of claim 1, comprising directing, by theprocessor, a machine-learning training component to train the system tomore accurately respond to inputs that share a characteristic with theinput dataset, wherein the more accurate response to inputs that sharethe characteristic with the input dataset comprises more accuratelyidentifying, explaining, and mitigating potential inconsistenciesbetween modules of the application.
 9. A computer-implemented method,the method comprising: in response to a submission of an input datasetto an artificially intelligent application, receiving an explanationfrom each module of the application, wherein the modules are configuredwithin the application in a serial sequence in which each module, uponreceiving the input dataset and any input generated by an immediatelypreceding module of the serial sequence, generates output that isforwarded as input to a next module, if any, in the sequence; anddetermining that at least two of the received explanations aresemantically inconsistent.
 10. The method of claim 9, wherein thedetermining comprises: identifying one or more potentially inconsistentpairs of explanations, of the received explanations, by performing asemantic analysis upon each explanation of the received explanations;and filtering out a false-positive pair of the potentially inconsistentpairs by using an artificially intelligent method of cognitive computingto determine that inconsistencies between two explanations of thefalse-positive pair are an expected consequence of a characteristic ofthe input dataset.
 11. The method of claim 10, wherein the more accurateresponse to inputs that share the characteristic with the input datasetcomprises more accurately identifying the false-positive pair.
 12. Themethod of claim 9, wherein the determining comprises: quantifying adegree of inconsistency between the two semantically inconsistentexplanations as a function of a total number of elements of the inputdataset that are associated with a Boolean inconsistency between a firstexplanation of the two semantically inconsistent explanations and asecond explanation of the two semantically inconsistent explanations;and identifying that the degree of inconsistency exceeds a predefinedthreshold level.
 13. The method of claim 9, wherein the determiningcomprises: determining an existence of a systemic error in one or bothof two modules of the application in response to a determination that acorresponding element of an inconsistency vector exceeds a predefinedthreshold inconsistency value, wherein each element of the inconsistencyvector identifies a relative number of elements of the input datasetthat are associated with a Boolean inconsistency between theexplanations generated by the two modules.
 14. The method of claim 13,comprising: recording a log entry in an inconsistency log, wherein thelog entry identifies the modules corresponding to the two semanticallyinconsistent explanations and the input dataset; forwarding the log to amachine-learning training component; and directing the machine-learningtraining component to incorporate contents of the log entry into amachine-learning corpus to be submitted to the application during asubsequent machine-learning training session.
 15. The method of claim 9,comprising providing at least one support service for a purpose selectedfrom the group consisting of: creating, integrating, hosting,maintaining, and deploying computer-readable program code in a computersystem, wherein the computer-readable program code in combination withthe computer system is configured to implement the receiving and thedetermining.
 16. A computer program product, the computer programproduct comprising a computer readable storage medium having programinstructions embodied therewith, the program instructions beingexecutable by a processor to cause the processor to perform a methodcomprising: receiving, by the processor, an explanation from each moduleof an artificially intelligent application, the explanations beingcreated by the modules in response to a submission of an input datasetto the application, wherein the modules are configured within theapplication in a serial sequence in which each module, upon receivingthe input dataset and any input generated by an immediately precedingmodule of the serial sequence, generates output that is forwarded asinput to a next module, if any, in the sequence; and determining, by theprocessor, that at least two of the received explanations aresemantically inconsistent.
 17. The computer program product of claim 16,wherein the determining comprises: identifying one or more potentiallyinconsistent pairs of explanations, of the received explanations, byperforming a semantic analysis upon each explanation of the receivedexplanations; and filtering out a false-positive pair of the potentiallyinconsistent pairs by using an artificially intelligent method ofcognitive computing to determine that inconsistencies between twoexplanations of the false-positive pair are an expected consequence of acharacteristic of the input dataset.
 18. The computer program product ofclaim 17, wherein the more accurate response to inputs that share thecharacteristic with the input dataset comprises more accuratelyidentifying the false-positive pair.
 19. The computer program product ofclaim 16, wherein the determining comprises: quantifying a degree ofinconsistency between the two semantically inconsistent explanations asa function of a total number of elements of the input dataset that areassociated with a Boolean inconsistency between a first explanation ofthe two semantically inconsistent explanations and a second explanationof the two semantically inconsistent explanations; and identifying thatthe degree of inconsistency exceeds a predefined threshold level. 20.The computer program product of claim 16, wherein the determiningcomprises: determining an existence of a systemic error in one or bothof two modules of the application in response to a determination that acorresponding element of an inconsistency vector exceeds a predefinedthreshold inconsistency value, wherein each element of the inconsistencyvector identifies a relative number of elements of the input datasetthat are associated with a Boolean inconsistency between theexplanations generated by the two modules.